describe and visualize of US weather event data

city selection - multivariate

purpose of notebook

(-) describe & visualize relationship between variables (multivariate) (-) correlation plots (-) gather interesting observations for further investigation (-) gather possible new features for extraction

todos: (-) …

information
https://www.kaggle.com/sobhanmoosavi/us-weather-events https://smoosavi.org/datasets/lstw
Moosavi, Sobhan, Mohammad Hossein Samavatian, Arnab Nandi, Srinivasan Parthasarathy, and Rajiv Ramnath. “Short and Long-term Pattern Discovery Over Large-Scale Geo-Spatiotemporal Data.” In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, 2019.
Weather event is a spatiotemporal entity, where such an entity is associated with location and time. Following is the description of available weather event types:
Severe-Cold: The case of having extremely low temperature, with temperature below -23.7 degrees of Celsius. Fog: The case where there is low visibility condition as a result of fog or haze. Hail: The case of having solid precipitation including ice pellets and hail. Rain: The case of having rain, ranging from light to heavy. Snow: The case of having snow, ranging from light to heavy. Storm: The extremely windy condition, where the wind speed is at least 60 km/h. Other Precipitation: Any other type of precipitation which cannot be assigned to previously described event types.
The weather data is provided in terms of a CSV file with the following attributes: Attribute Description Nullable 1 EventId This is the identifier of a record No 2 Type The type of an event; examples are rain and snow. No 3 Severity The severity of an event, wherever applicable. Yes 4 StartTime (UTC) The start time of an event in UTC time zone. No 5 EndTime (UTC) The end time of an event in UTC time zone. No 6 TimeZone The US-based timezone based on the location of an event No (eastern, central, mountain, and pacific). 7 LocationLat The latitude in GPS coordinate. Yes 8 LocationLng The longitude in GPS coordinate. Yes 9 AirportCode The airport station that a weather event is reported from. Yes 10 City The city in address record. Yes 11 County The county in address record. Yes 12 State The state in address record. Yes 13 ZipCode The zipcode in address record. Yes

insights

1. spearman correlation is rang based and needs at least ordinal scale, thus Type and City are out there is a small positive correlation between StartTime and Severity, since StartTime is mostly the increase in time over the years, thus Severity might gone up a little
2. in Severity there are some gaps eg Rain and Snow have no Severe, where Fog has only Severe and Moderate, Cold and Storm come only in Severe, Hail only has Other these gaps make things hard to compare in Light Rain is most, but already in Moderate there is more Fog (because of Los Angeles)
3. Fog seems to be longer than Rain, especially in Houston and Los Angeles; Los Angeles has no Snow but longer Cold periods (comparable to New York); Snow longest in New York and Seattle, but Seattle has less Cold, overall quite some severe fog overall
4. Light and Moderate events are the most, there are less Severe events but there are also as long; Severe events in Houston and Seattle are quite long, Moderate is longer than usual in Los Angeles; may be most are Fog

stories

(!) compare 4 big cities which are all close to the coast but vary in significantly in longitude and latitude

load packages

overview

head(weather)

summary(weather)

   EventId                     Type          Severity      StartTime                      EndTime                          TimeZone     LocationLat     LocationLng     
 Length:12226       Cold         : 365   Heavy   : 439   Min.   :2016-01-01 08:47:00   Min.   :2016-01-01 09:47:00   US/Central:2900   Min.   :29.98   Min.   :-122.31  
 Class :character   Fog          :2920   Light   :6792   1st Qu.:2017-04-08 08:59:45   1st Qu.:2017-04-08 11:06:30   US/Eastern:3656   1st Qu.:34.02   1st Qu.:-122.31  
 Mode  :character   Hail         :   6   Moderate:3428   Median :2018-09-27 02:42:30   Median :2018-09-27 03:02:30   US/Pacific:5670   Median :40.78   Median : -95.36  
                    Precipitation: 139   Other   :   6   Mean   :2018-08-09 22:18:52   Mean   :2018-08-09 23:49:14                     Mean   :38.70   Mean   :-100.68  
                    Rain         :8296   Severe  :1422   3rd Qu.:2019-11-24 10:37:30   3rd Qu.:2019-11-24 12:37:30                     3rd Qu.:47.44   3rd Qu.: -73.97  
                    Snow         : 494   UNK     : 139   Max.   :2020-12-31 23:22:00   Max.   :2020-12-31 23:53:00                     Max.   :47.44   Max.   : -73.97  
                    Storm        :   6                                                                                                                                  
          City        Duration       
 Houston    :2900   Length:12226     
 Los Angeles:2378   Class :difftime  
 New York   :3656   Mode  :numeric   
 Seattle    :3292                    
                                     
                                     
                                     

correlation
spearman correlation is rang based and needs at least ordinal scale, thus Type and City are out there is a small positive correlation between StartTime and Severity, since StartTime is mostly the increase in time over the years, thus Severity might gone up a little
```r cor_all <- weather %>% # select(Type, Severity, StartTime, City, Duration) %>% select(Severity, StartTime, Duration) %>% # mutate(Type = as.numeric(Type)) %>% mutate(Severity = as.numeric(Severity)) %>% mutate(StartTime = as.numeric(StartTime)) %>% # mutate(City = as.numeric(City)) %>% mutate(Duration = as.numeric(Duration)) %>% cor(method = ‘spearman’)
corrplot(cor_all) ```

multivariate Severity over Type

in Severity there are some gaps eg Rain and Snow have no Severe, where Fog has only Severe and Moderate, Cold and Storm come only in Severe, Hail only has Other these gaps make things hard to compare in Light Rain is most, but already in Moderate there is more Fog (because of Los Angeles)

type_severity_plot_point <- weather %>%
  ggplot(aes(x = fct_infreq(Type), y = fct_infreq(Severity), col = fct_infreq(City))) +
    geom_jitter(alpha = 0.2, size = 0.5) +
    theme_minimal() +
    ggtitle("Severity over Type")
  
ggplotly(type_severity_plot_point)  

type_severity_plot_point <- weather %>%
  filter(as.numeric(Duration) < 600) %>%
  ggplot(aes(x = fct_infreq(Type), y = fct_infreq(Severity), alpha = as.numeric(Duration))) +
    geom_jitter(size = 0.5) +
    theme_minimal() + 
    # facet_grid(vars(City)) +
    ggtitle("Severity over Type and Duration as alpha")
  
ggplotly(type_severity_plot_point)  

# mosaic plot from https://cran.r-project.org/web/packages/ggmosaic/vignettes/ggmosaic.html
type_severity_plot_mosaik <-weather %>%
  mutate(Severity = fct_infreq(Severity)) %>% # order Severity in frequency
  mutate(Type = fct_infreq(Type)) %>% # order Type in frequency
  ggplot() +
    geom_mosaic(aes(x = product(Type, Severity), fill=City)) + # little better to see without fill=City, fct_infreq() doesn't work here
    labs(title='f(Type | Severity) f(City)') +
    theme_minimal() +
    ggtitle("Severity over Type")

ggplotly(type_severity_plot_mosaik)

multivariate Duration over Type

Fog seems to be longer than Rain, especially in Houston and Los Angeles; Los Angeles has no Snow but longer Cold periods (comparable to New York); Snow longest in New York and Seattle, but Seattle has less Cold, overall quite some severe fog overall

duration_type_plot_point <- weather %>%
  ggplot(aes(x = fct_infreq(Type), y = as.numeric(Duration), col = fct_infreq(City))) +
    geom_jitter(alpha = 0.2, size = 0.5) +
    theme_minimal() +
    ggtitle("Duration over Type")

ggplotly(duration_type_plot_point)

duration_type_plot_point <- weather %>%
  filter(as.numeric(Duration) < 600) %>%
  ggplot(aes(x = fct_infreq(Type), y = as.numeric(Duration), col = fct_infreq(City))) +
    geom_boxplot() +
    theme_minimal() +
    facet_grid(rows = vars(City)) +
    # coord_flip() +
    ggtitle("Duration over Type")

ggplotly(duration_type_plot_point)

multivariate Duration over Severity

Light and Moderate events are the most, there are less Severe events but there are also as long; Severe events in Houston and Seattle are quite long, Moderate is longer than usual in Los Angeles; may be most are Fog

duration_severity_plot_point <- weather %>%
  ggplot(aes(x = fct_infreq(Severity), y = as.numeric(Duration), col = fct_infreq(City))) +
    geom_jitter(alpha = 0.2, size = 0.5) +
    theme_minimal() +
    ggtitle("Severity over Type")

ggplotly(duration_severity_plot_point)

duration_severity_plot_box <- weather %>%
  filter(as.numeric(Duration) < 600) %>%
  ggplot(aes(x = fct_infreq(Severity), y = as.numeric(Duration), col = fct_infreq(City))) +
    geom_boxplot() +
    theme_minimal() +
    facet_grid(rows = vars(City)) +
    # coord_flip() +
    ggtitle("Duration over Severity")

ggplotly(duration_severity_plot_box)